Carry the division remainder into rounding as a sticky bit (#236)#237
Open
b-erdem wants to merge 1 commit into
Open
Carry the division remainder into rounding as a sticky bit (#236)#237b-erdem wants to merge 1 commit into
b-erdem wants to merge 1 commit into
Conversation
Decimal.div/2 computed the quotient to precision+1 digits and then
discarded the long-division remainder, so the rounding step could not
tell an exact tie from a value with more nonzero digits below the guard
digit. This rounded ~5% of inexact divisions the wrong way:
* :half_even / :half_down — a guard digit of 5 with a nonzero tail is
not a tie and must round up, e.g. 4.7460 / -5522 at precision 28
truly ends …1405287…; it returned …140 instead of …141.
* :ceiling / :floor / :up — a guard digit of 0 with a nonzero tail is
still nonzero, so :ceiling could return a result below the true
quotient (and :floor above it), violating the modes' guarantee.
Pass `rem != 0` as the sticky bit to context/3, which every rounding
mode's increment?/5 already consumes. This also corrects the :inexact
flag, which context_test.exs pinned as suppressed when the remainder was
a power of ten; both :rounded and :inexact are now signalled, matching
the spec and Python's decimal.
Verified with a differential fuzz against Python's decimal: all 7
rounding modes, 42,000 random divisions, 0 mismatches after the fix
(previously ~5% per mode).
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the
:half_even/:ceiling/:floorrounding half of #236.Problem
Decimal.div/2computes the quotient toprecision + 1digits and then discards the long-division remainder, so the rounding step cannot distinguish an exact tie from a value with more nonzero digits below the guard digit. This rounds roughly 5% of inexact divisions the wrong way::half_even/:half_down— a guard digit of 5 with a nonzero tail is not a tie and must round up.4.7460 / -5522at precision 28 truly ends…1405287…; decimal returned…140instead of…141.:ceiling/:floor/:up— a guard digit of 0 with a nonzero tail is still nonzero.:ceilingcould return a result below the true quotient (and:floorabove it), violating the modes' defining guarantee. Example:Decimal.div(Decimal.new("1.0000001"), Decimal.new("1"))at precision 5:ceilingreturned1.0000(below the true value) instead of1.0001.Fix
div_calc/5already returns the remainder; this passesrem != 0tocontext/3as the sticky bit, which every rounding mode'sincrement?/5already consumes (the same mechanismadd/2uses since the CVE‑2026‑32686 mitigation). One line.The
:inexactflagThe same remainder also feeds the
:inexactcondition.context_test.exsasserted[:rounded](not:inexact) for10^106 / 17at precision 111, because:inexactwas suppressed whenever the remainder happened to be a power of ten. That division is non-terminating, so the rounded result is inexact — Python'sdecimalflags bothRoundedandInexact. The test is updated to assert both.Verification
Differential fuzz against Python's
decimal(which passes the official GDAdecarithsuite): all 7 rounding modes × 6,000 random divisions each (42,000 total), 0 mismatches after the fix — every result adjudicated against the exact quotient with rational arithmetic. Before the fix, ~5% per mode diverged.Tests
decimal_test.exs— newdiv/2regression test covering the guard‑5 sticky case, genuine ties still rounding to even, and the:ceiling/:floordirectional cases.context_test.exs— the10^106 / 17flag assertion now expects:roundedand:inexact.All existing tests pass (
101 doctests, 27 properties, 102 tests).